Generation of robotic user interface responsive to connection of peripherals to robot

ABSTRACT

Methods and systems for connection-driven generation of robotic user interfaces and modification of robotic properties include detecting a connection of a robotic peripheral to a robot; obtaining a peripheral property set corresponding to the robotic peripheral, wherein the peripheral property set includes one or more properties of the robotic peripheral; modifying, based on the peripheral property set, a robotic property set that includes one or more properties of the robot to provide a modified robotic property set; generating, during runtime, a robotic graphical user interface (“RGUI”) dynamically based on the peripheral property set, wherein the RGUI provides at least one user-accessible interface to control the robot and the robotic peripheral; and controlling, based on the modified robotic property set, the robot and the robotic peripheral in response to user input received via the RGUI.

CLAIM OF PRIORITY

This application is a continuation of U.S. Pat. No. 11,097,415 filed onJul. 24, 2019, which is a continuation of U.S. Pat. No. 10,399,220 filedon Nov. 2, 2016, which claims priority to U.S. Provisional PatentApplication 62/249,858 filed on Nov. 2, 2015, the disclosures of whichare hereby incorporated by reference in their entireties.

GOVERNMENT SUPPORT STATEMENT

This invention was made with Government support under Grant No.NRI-1227277 awarded by the National Science Foundation. The U.S.Government has certain rights in this invention.

FIELD

The present disclosure relates generally to systems and methods thatdynamically generate and adapt robotic user interfaces, with which userscan interact with and program at least one robot, based on a roboticconfiguration of the robot and changes in the robotic configuration.

BACKGROUND

Robotic industrial automation has seen significant success inlarge-scale manufacturing because it offers significant advantages atscale for tasks such as welding, cutting, stamping, painting, heavymaterial handling, precision material machining, etc. The success ofrobotic automation in large-scale manufacturing has led to along-standing desire to extend the use of robotic automation into smalland medium-sized manufacturing enterprises (“SMEs”). However, incontrast to large scale manufacturing, SMEs' production processes aretypically characterized by small production volumes and/or high productvariability. Consequently, the ability to amortize the infrastructure,specialized personnel, setup, and programming of flexible roboticautomation is far reduced for SMEs.

SME processes sometimes include tasks that require a high level ofcustomization and therefore necessarily involve human skill andjudgment. For example, refurbishment tasks and build-to-ordermanufacturing processes must accommodate unforeseen workpiece variancesand equipment modifications. In such cases, an existing human-centeredproduction process may find it difficult to determine where or howrobotic automation can be a useful addition to an effectivehuman-intensive process, rather than a duplication or attenuationthereof. Take, for instance, an SME specializing in custom furnituremanufacturing that has a number of highly-skilled employees. That SMEmay want to improve the efficiency and productivity of its employees byusing robotic systems to automate repetitive tasks that involvedexterous actions, such as drilling or sanding tasks. However, acommercial off-the-shelf robotic system would not be useful in this casebecause it would be impossible for the SME to leverage its employees'existing task knowledge and experience.

User experience associated with expanding capability of robotic systemscan also be problematic during robotic system integration. When a robotis being equipped or outfitted with additional devices (e.g., sensors,cameras, grippers, etc.) or end effector tooling, there is a requirementthat programming effort must be made for these devices or tooling to beutilized by the end user. For example, an out-of-the-box robotic systemmay include a programming language interface (“PLI”) for moving the armaround in Cartesian space. If a gripper is attached to the robot to giveit the ability to grasp objects, a user (e.g., an integrator) mustinstall firmware for this gripper, which then exposes some set ofprogrammatic entities to the PLI. Now consider the example where thesame robotic system incorporates a graphical user interface (“GUI”) forprogramming. The gripper is attached and firmware must be installed toprovide the appropriate GUI entities so that the user can access therobotic system's functionality. For most state of the art systems, thisseamless integration of an appropriate GUI for attached tooling is notpossible. Generally, if this type of functionality is desired, thesystem is built from the ground up with the GUI for the tooling, withoutany accommodation for expandability.

Therefore, there is a need for systems and methods for overcoming theseand other problems presented by the prior art.

SUMMARY

There is increasing interest in more flexible robotic systems andplatforms, particularly for SMEs. This includes robots that may haveeasy-to-install or hot-swappable robotic peripherals, such as sensors,end effectors, tooling, and other peripherals. There is also anincreased interest in the ease-of-use of these systems for noviceend-users. As GUI-based systems for programming robots become morewidespread, there are emergent needs for automatic generation of dynamicGUI elements and modification of robotic properties responsive toon-line integration of hot-swappable hardware or robotic peripheral.

Various embodiments of the present disclosure include examples of aGUI-based robotic programming platform that obtains a roboticconfiguration of one or more robots, detects changes in the roboticconfiguration, and adaptively generates robotic GUIs (“RGUIs”) andupdates robotic properties of the robots based on the roboticconfiguration and the changes in the robotic configuration. The RGUIsprovide interfaces for users to interact with or program the robotsand/or one or more robotic modules connected to the robots.

Embodiments of the present disclosure relate to systems and methods forperforming connection-driven modification of robots' properties andgeneration of RGUIs for the robots in response to robotic peripheralsbeing connected to or disconnected from the robots. Specifically, acomputer-implemented method for connection-driven generation of RGUIsand modification of robotic properties includes detecting a connectionof a robotic peripheral to a robot; obtaining a peripheral property setcorresponding to the robotic peripheral, wherein the peripheral propertyset includes one or more properties of the robotic peripheral;modifying, based on the peripheral property set, a robotic property setthat includes one or more properties of the robot to provide a modifiedrobotic property set; generating, during runtime, a RGUI dynamicallybased on the peripheral property set, wherein the RGUI provides at leastone user-accessible interface to control the robot and the roboticperipheral; and controlling, based on the modified robotic property set,the robot and the robotic peripheral in response to user input receivedvia the RGUI.

The GUI-based robotic programming platform examples provide capabilitiesfor seamless on-line swapping of robotic hardware and software moduleson robotic systems, with minimal to no requirement for integratingdrivers that operate such robotic modules or GUI elements that allowusers to interact with and control such robotic modules. Additionalobjects and advantages of the embodiments of the disclosure will be setforth in part in the description which follows, and in part will beobvious from the description, or may be learned by practice of theembodiments. The objects and advantages of the embodiments will berealized and attained by means of the elements and combinationsparticularly pointed out in the appended claims.

It is to be understood that both the foregoing general description andthe following detailed description are exemplary and explanatory onlyand are not restrictive of the embodiments, as claimed.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates an embodiment of a robotic programming platform thatdynamically generates and adapts RGUIs for one or more robots based onthe robots' configuration and changes therein, consistent with theprinciples of the present disclosure.

FIG. 2 is a flow diagram illustrating an example of a method fordynamically generating and adapting RGUIs for one or more robots basedon the robots' configuration and changes therein, consistent withembodiments of the present disclosure.

FIGS. 3, 4A, 4B, and 5 illustrate an example scenario in which therobotic programming platform dynamically generates and adapts RGUIs fora robot based on the robot's configuration and changes therein,consistent with the principles of the present disclosure.

FIG. 6 is an example computer system for performing the disclosedembodiments, consistent with the present disclosure.

DETAILED DESCRIPTION

Reference will now be made in detail to example embodiments, which areillustrated in the accompanying drawings. When appropriate, the samereference numbers are used throughout the drawings to refer to the sameor like parts.

For simplicity and illustrative purposes, the principles of the presentdisclosure are described by referring mainly to exemplary embodimentsthereof. However, one of ordinary skill in the art would readilyrecognize that the same principles are equally applicable to, and can beimplemented in, all types of information and systems, and that any suchvariations do not depart from the true spirit and scope of the presentdisclosure. Moreover, in the following detailed description, referencesare made to the accompanying figures, which illustrate specificexemplary embodiments. Electrical, mechanical, logical and structuralchanges may be made to the exemplary embodiments without departing fromthe spirit and scope of the present disclosure. The following detaileddescription is, therefore, not to be taken in a limiting sense and thescope of the present disclosure is defined by the appended claims andtheir equivalents.

In the present disclosure, the word “robot” will be used instead ofrobotic manipulator or robotic system. Typically, a robot is anindustrial robotic manipulator or a set of industrial roboticmanipulators for automated or semi-automated production applications. Arobot's end effector can be a tool effector attached to the robot at aconnection site or a tool grasped or held by a gripper-type end effectorattached to the robot. The following general definitions will be usedherein. Robotic hardware module: a physical device or tooling that canbe connected (e.g., attached, coupled, linked, etc.) to a robot and/ordisconnected (e.g., detached, decoupled, delinked, and the like) fromthe robot. Peripheral or robotic peripheral: a robotic hardware modulethat can be connected to a robot to provide some physical interaction,manipulation, and/or sensing capability. Active peripheral: a peripheralthat can communicate with a robot via one or more communication channelssuch as a network connection, a Universal Serial Bus (“USB”) connection,etc., and may or may not have a physical connection with the robot.Passive peripheral: a peripheral with only a mechanical connection to arobot. Software module: a unit of software that encapsulates someprocessing, algorithmic, or inference capability accessible via at leastone defined interface. Software component: an instantiated softwaremodule. User interface (“UI”): a visual, gestural, haptic, or audiointerface that allows for user interaction with a robot. Roboticgraphical UI (“RGUI”): at least one UI that allows a user to program orinteract with a robot using graphical widgets on a supported device. UIcomponent: a widget or graphical element that can accept input andevents from a user and provide information to the user.

Various embodiments of the present disclosure include systems andmethods for adaptively generating RGUIs and updating robotic propertiesbased on a robotic configuration and changes in the roboticconfiguration. An exemplary robotic programming platform obtains arobotic configuration of one or more robots, generates RGUIs for therobots adaptively based on the robots' robotic configuration, detectschanges in the robots' robotic configuration, and adapts the RGUIs forthe robots dynamically in response to the detected changes in therobotic configuration. The robotic programming platform utilizes eventmonitors, user input, and/or sensors to detect changes in the robots'robotic configuration, which can result from, for example, robotichardware modules being connected to or disconnected from the robots,robotic software modules being enabled or disabled, a subset of robotichardware modules connected to the robots or robotic software modulesenabled for the robots being selected or deselected, or the like.

The robotic programming platform dynamically adds or adapts GUI elementsto adaptively generate one or more RGUIs for at least one user tointeract with or program the robots and/or one or more robotic modulesconnected to the robots. The robotic programming platform can (1)dynamically add or adapt appropriate GUI elements for the RGUIs inresponse to robotic hardware modules being physically connected to therobots; (2) dynamically add or adapt appropriate GUI elements in RGUIsin response to add-on robotic software modules being enabled; (3)dynamically load GUI elements in RGUIs in response to a subset ofconnected robotic hardware modules or enabled robotic software modulesbeing selected; (4) add or adapt, in real-time, GUI elements in RGUIs inresponse to system-sensed changes in the physical configuration of therobots or connected robotic hardware modules, and (5) adapt, inreal-time, GUI elements in RGUIs based on user-indicated changes to thephysical configuration of the robots or connected robotic hardwaremodules.

The robotic programming platform can also dynamically update roboticproperties of the robots in response to the detected changes in therobotic configuration. The robotic programming platform providescapabilities for seamless on-line swapping of robotic modules on therobots, with minimal to no requirement for the user to integrate driversthat operate such robotic modules and/or GUI elements that allow theuser to interact with and control such robotic modules. By factoring inthe robotic configuration, the changes in the robotic configuration,and/or the robotic properties of the robots, the robotic programmingplatform can adaptively customize the RGUIs to manage the user'sinteractions with the robots and/or the connected robotic modules. Forexample, the robotic programming platform can adaptively generate RGUIsthat grant the user with access to all, some, or none of functionalitiesof the robots and/or the connected robotic modules.

Examples of the robotic programming platform provide (1) the ability toconnect robotic hardware modules to one or more robots, which enablesthe automatic loading of drivers and UI components, as well as theupdating of the robots' physical properties; (2) the ability to connectrobotic software modules to the robots, which enables the automaticloading of drivers and UI components; and (3) the ability to connectrobotic hardware modules to the robots via non-physical connections(e.g., network or wired connections, fluid couplings, etc.) in such away that the robots can operate through various means (e.g.,electrically, pneumatically, hydraulically) or communicate with therobotic hardware modules, which enables the automatic loading of driversand UI components specific to the robotic hardware modules.

FIG. 1 illustrates an exemplary embodiment of a robotic programmingplatform that dynamically generates and adapts UIs for at least onerobot 50 based on a configuration of robot 50 and changes in theconfiguration of robot 50, consistent with the principles of the presentdisclosure. In various embodiments, an example of which is shown in FIG.1 , a platform 100 includes an attachment manager 110, which is asoftware component loaded at runtime. Attachment manager 110 provides anintegral attachment manager UI (“MUI”) 115, which a user can utilize toconfigure, program, or interact with robot 50 and/or one or more roboticmodules connected to robot 50. Platform 100 includes a UI manager 120,which is communicatively coupled to attachment manager 110 and can beinvoked to dynamically generate and adapt one or more RGUIs 130 based onthe robotic configuration of robot 50 and changes in the roboticconfiguration of robot 50.

Platform 100 includes a robotic properties data store 140 that specifiesone or more robotic properties of robot 50, examples of which includemass-related information, inertia-related information, dynamics-relatedinformation, collision-related information, control gains-relatedinformation, simulation-related information, logical state-relatedinformation, kinematics-related information, connection site-relatedinformation, one or more images representative of robot 50, and thelike. Robotic properties data store 140 is accessible to attachmentmanager 110 and can be stored locally therein. Attachment manager 110can obtain the robotic properties of robot 50 from robotic propertiesdata store 140, utilize the robotic properties to derive the roboticconfiguration of robot 50, and use attachment MUI 115 to display therobotic configuration of robot 50. For example, attachment manager 110can obtain an image representative of robot 50 and information relatedto one or more connection sites of robot 50, modify the image toindicate the connection sites' relative locations on robot 50, and useattachment MUI 115 to display the modified image of robot 50 thatvisually indicates the relative locations of the connection sites ofrobot 50.

Platform 100 includes an attachment manifest 150 that specifiesattachment parameters and properties related to all authorized roboticperipherals capable of being connected to robot 50. Attachment manifest150 catalogs one or more attachment parameters and properties related toeach robotic peripheral, examples of which include physical parametersand properties (e.g., the robotic hardware module's name or identifier,model, type, relative mounting offset, size, mass, connection sites,etc.), UI component requirements, driver requirements, one or morerepresentative images, and the like. Attachment manifest 150 isaccessible to attachment manager 110 and can be stored locally therein.Attachment manager 110 can look up and locate attachment perimeters andproperties related to a robotic peripheral based on the roboticperipheral's identifier, and if the robotic peripheral's identifiercannot be located in attachment manifest 150, attachment manager 150 canreject the robotic peripheral as unauthorized and prevent theunauthorized robotic peripheral from being connected to robot 50.

Attachment manager 110 detects changes in the robotic configuration ofrobot 50 by discovering or detecting one or more attachment eventsrelated to robot 50, determining that the attachment events change therobotic configuration of robot 50, and identifying how the attachmentevents change the robotic configuration of robot 50. Platform 100includes at least one event monitor 160, attachment MUI 115, and/orsensors (not shown) in platform 100 or robot 50, which attachmentmanager 110 can utilize to discover or detect attachment events such asrobotic peripherals being connected to robot 50 or disconnected fromrobot 50, robotic software modules being enabled or disabled for robot50, a subset of robotic peripherals connected to robot 50 or roboticsoftware modules enabled for robot 50 being selected or deselected, andthe like. Event monitor 160 monitors robot 50 and/or other applicableevent sources in platform 100 to discover or detect peripheralattachment events triggered by active peripherals being connected torobot 50 or disconnected from robot 50. For instance, event monitor 160can monitor one or more communication channels (e.g., network or USBconnections, etc.) for peripheral attachment events. In response to theuser connecting an active peripheral to robot 50, at least one of thecommunication channels dispatches a peripheral attachment event, whichcan include information related to the active peripheral, such as theactive peripheral's peripheral identifier. Event monitor 160 can receiveand forward the peripheral attachment event to attachment manager 110,which can process the peripheral attachment event to detect changes inthe robotic configuration of robot 50 resulting from the attachment ofthe active peripheral to robot 50. Attachment manager 110 can identifythe active peripheral and/or obtain information related to the activeperipheral, for example, by processing the peripheral attachment event,polling robot 50 and/or the active peripheral, querying attachmentmanifest 150 and/or the user via attachment MUI 115, or the like.

When the user connects a passive peripheral to robot 50 or disconnects apassive peripheral from robot 50, the user can utilize attachment MUI115 to identify the passive peripheral and specify that the passiveperipheral has been connected to robot 50 or disconnected from robot 50.To identify the passive peripheral, the user can use attachment MUI 115to input or select a peripheral identifier corresponding to the passiveperipheral. In response, attachment MUI 115 dispatches a user-specifiedattachment event to attachment manager 110, which can process theuser-specified attachment event to detect changes in the roboticconfiguration of robot 50 resulting from the attachment or detachment ofthe passive peripheral to robot 50. Attachment manager 110 can identifythe passive peripheral and/or obtain information related to the passiveperipheral, for example, by processing the user-specified attachmentevent, polling robot 50, querying attachment manifest 150 and/or theuser via attachment MUI 115, or the like. Similarly, when the userenables or disables robotic software modules for robot 50, select ordeselect a subset of robotic peripherals connected to robot 50 orrobotic software modules enabled for robot 50, etc., attachment MUI 115can dispatch one or more user-specified attachment events to attachmentmanager 110, which can process the user-specified attachment events todetect changes in the robotic configuration of robot 50.

The user can use attachment MUI 115 to view and specify the physicalconfiguration of the robotic peripheral relative to robot 50, includingwhether or not the robotic peripheral is physically connected to robot50, and if physically connected, then at which connection site of robot50, or if not physically connected, whether or not the roboticperipheral is connected to robot 50 via a non-physical connection (e.g.,network or wired connections, fluid couplings, etc.). Attachment MUI 115includes a drag-and-drop interface for specifying which roboticperipherals are attached to robot 50 and where, and can display one ormore robotic properties of robot 50. For example, attachment manager 110can display one or more connection sites of robot 50 capable ofaccepting the robotic peripheral. Attachment manager 110 tracks a set ofcurrently-connected robotic peripherals that are connected to robot 50and denote the set as the loadout of robot 50.

When the user utilizes attachment MUI 115 to drag a peripheral iconrepresentative of the robotic peripheral to a robot icon representativeof robot 50, attachment MUI 115 can highlight any of the connectionsites of robot 50 that can mechanically accept the robotic peripheral,as defined by attachment manifest 150. The specification on how thisattachment takes place, including the relative mounting offsets of thedifferent components is captured in attachment manifest 150. Thiscreates a seamless experience for the user where they do not have tospecify offsets directly. The robotic peripherals, assuming theirmanifest in attachment manifest 150 contains the appropriate attachmentinformation, will “snap” to the proper location on the robot.

TABLE 1 An Example Configuration File Property Type Property Value name:Ready Adaptive Gripper model: RG-A info: Robotiq Type 85 2-FingerGripper mesh: ready_adaptive_gripper.STL image:ready_adaptive_gripper.png urdf: ready_adaptive_gripper.urdf.xacromacro_name: ready_adaptive_gripper attachment_link:ready_adaptive_base_link offset_xyz: [0, 0, 0] offset_rpy: [1.5707, 0,1.5707] type: adaptive_gripper driver: [robotiq_c_model_control,CModelTeleop.launch] ui: [ready_adaptive_gripper_open, ready_adaptive_gripper_close]

When a loadout has been defined, the user can trigger attachment manager110 to regenerate, which includes several actions: (a) attachmentmanager 110 looks through attachment manifest 150 for at least oneconnected robotic peripheral and match any that have been specified inthe loadout; (b) attachment manager 110 updates the current roboticproperties file with the mass, inertial or dynamic information,collision properties or other properties of the connected roboticperipheral; (c) attachment manager 110 loads drivers required by theconnected robotic peripheral as specified in attachment manifest 150;and (d) attachment manager 110 communicates with any UI components inplatform 100 or RGUIs 130, and load the required UIs for the connectedrobotic peripheral. Attachment manager 110 can utilize a runtime drivermanager 180 to search drivers 190 for drivers required by the connectedrobotic peripheral and load the required drivers, as well as unload aloaded robotic peripheral driver. Attachment manager 110 can utilizeuser interface manager 120 to load or unload UI components and RGUIs 130required or specified by the connected robotic peripheral.

FIG. 2 illustrates an example process for dynamically generating andadapting RGUIs for one or more robots based on the robots' configurationand changes therein, consistent with embodiments of the presentdisclosure. Process 200 is performed in platform 100. More particularly,attachment manager 110 in platform 100 performs process 200 to detectchanges in a robotic configuration of robot 50 and dynamically generateand adapt RGUIs for robot 50. Process 200 starts at stage 210, at whichattachment manager 110 obtains robotic properties and a roboticconfiguration of robot 50. Attachment manager 110 defines a loadout ofrobot 50 based on the robotic properties and/or configuration of robot50 and/or any robotic peripherals connected to robot 50. Attachmentmanager 110 can generate a visual representation of robot 50 and/or anyconnected robotic peripherals based on the loadout and display thevisual representation via attachment MUI 115. For example, attachmentMUI 115 can display the visual representation of robot 50 thathighlights one or more connection sites of robot 50.

At stage 210, attachment manager 110 monitors robot 50 and/or platform100 to discover or detect attachment events related to robot 50. Moreparticularly, attachment manager 110 can detect or discover roboticperipherals being connected to or disconnected from robot 50, roboticsoftware modules being enabled or disabled for robot 50, a subset ofconnected robotic peripherals or enabled robotic software modules beingselected or deselected, and the like. In response to discovering ordetecting at least one attachment event related to robot 50, attachmentmanager 110 identifies how the attachment event changes the roboticconfiguration of robot 50, by classifying the attachment event as eitherperipheral-related or not peripheral-related and then modifying therobotic configuration of robot 50 accordingly. If the attachment eventis classified at stage 200 as peripheral-related, i.e., caused by aperipheral being connected to robot 50, then process 200 proceeds tostage 220.

At stage 220, attachment manager 110 classifies the peripheral as anactive peripheral or a passive peripheral. If at stage 220 theperipheral is classified as active, then process 200 proceeds to stage230, at which attachment manager 110 obtains or polls the activeperipheral and/or robot 50 for information related to the activeperipheral and/or the attachment event. Attachment manager 110 then usesthe peripheral-related information to look up the active peripheral'smanifest in attachment manifest 150 for parameters and propertiesrelated to the active peripheral. Attachment manager 110 can modify orupdate the robotic configuration of robot 50 based on the activeperipheral's parameters and properties, and then use attachment manager110 to regenerate the visual representation of robot 50 based on theupdated robotic configuration. Next, at stage 232, attachment manager110 loads drivers required by the active peripheral's manifest. Then, atstage 234, attachment manager 110 loads UI components required by theactive peripheral's manifest to dynamically generate and adapt RGUIs forrobot 50 and the active peripheral. Subsequent to stage 234, process 200returns to stage 210.

Alternatively, if at stage 220 the peripheral is classified as passive,then process 200 proceeds to stage 240, at which attachment manager 110obtains, via attachment manager UI 115, user-specified informationrelated to the passive peripheral and/or the attachment event. At stage242, attachment manager 110 then uses the user-specified information tolook up the passive peripheral's manifest in attachment manifest 150 forparameters and properties related to the passive peripheral. Attachmentmanager 110 can customize the visual representation of robot 50 based onthe passive peripheral's parameters and properties and display thecustomized visual representation via attachment MUI 115. For example,attachment MUI 115 can customize the visual representation of robot 50to indicate which, if any, of the connection sites of robot 50 iscompatible with the passive peripheral, e.g., can accept the passiveperipheral. Attachment manager 110 can receive user input via attachmentMUI 115 to select a compatible connection site for the passiveperipheral, and modify or update the robotic configuration of robot 50based on the selected connection site and passive peripheral'sparameters and properties. Attachment MUI 115 can then regenerate thevisual representation of robot 50 based on the updated roboticconfiguration. Next, at stage 244, attachment manager 110 loads driversrequired by the passive peripheral's manifest. Then, at stage 246,attachment manager 110 loads UI components required by the passiveperipheral's manifest to dynamically generate and adapt RGUIs for robot50 and the passive peripheral. Subsequent to stage 246, process 200returns to stage 210.

If at stage 210 the attachment event is classified as notperipheral-related, then process 200 proceeds to stage 250. At stage250, attachment manager 110 loads at least one robotic software modulesand then proceeds to stages 252 and 254, in which attachment manager 110dynamically generates and adapts RGUIs for robot 50 by looking up therobotic software module's manifest in attachment manifest 150, loadingdrivers required by the robotic software module's manifest, and loadingUI components required by the robotic software module's manifestSubsequent to stage 254, process 200 returns to stage 210.

FIG. 3 shows an example scenario 300, where a robot 310 has at least oneend effector 315 through which robot 310 can connect to and control atleast one robotic peripheral 330. End effector 315 serves as aconnection site of robot 310. As shown in FIG. 3 , at least one user canattach robotic peripheral 330 to end effector 315 to physically and/orelectrically connect robotic peripheral 330 to robot 310. In examplescenario 300, end effector 315 of robot 310 starts with no roboticperipherals. The user attaches ADAPTIVE_GRIPPER (an active peripheral)to end effector 315. The user then uses attachment MUI 115 to specifythat the gripper has been attached to end effector 315 of robot 310 withCOLLAR_A (a passive peripheral), by first dragging a graphicalrepresentation of COLLAR_A to the end effector location on a graphicalrepresentation of robot 310, then dragging the graphical representationof ADAPTIVE_GRIPPER to the end effector location. When the user dragsCOLLAR_A to the end of effector location of the graphical representationof robot 310, attachment MUI 115 checks that end effector 315 is aproperly defined attachment location for the COLLAR_A, and if so, thegraphical representation of COLLAR_A will “snap” to the end effectorlocation of the graphical representation of robot 310 at thepre-specified offsets contained in attachment manifest of COLLAR_A. Theuser then selects a “generate” button, and attachment manager 110 firstupdates the physical properties of robot 310 based on the new mass,inertial characteristics and collision parameters of COLLAR_A andADAPTIVE_GRIPPER. For instance, since the gripper and collar areconnected to end effector 315, robot 310 now has more weight on it, andtherefore attachment manager 110 can apply an offset to the gravitycompensation or force sensing components. Additionally, the end of robot310 now protrudes further, so the collision information is updated aswell. Attachment manager 110 then loads the driver for ADAPTIVE_GRIPPER,as the only active peripheral connected. Attachment manager 110 thenlooks to the attachment manifest for ADAPTIVE_GRIPPER, and finds twouser interface components (“UICs”) that must be loaded. Attachmentmanager 110 then makes these two UICs available to the rest of platform100, and any running RGUIs that accept components of that type willautomatically load the UICs. This will display to the user as two newbuttons for controlling the ADAPTIVE_GRIPPER in whichever user interfacethey are using. When the user changes out ADAPTIVE_GRIPPER for anothergripper PRECISION_GRIPPER, attachment manager 110 unloads any runningdrivers and UICs for ADAPTIVE_GRIPPER, and then loads the requireddrivers and UICs for PRECISION_GRIPPER.

Attachment manager 110 can also manage connected software components inthe same manner as robotic peripherals, where the user specifies that asoftware component has been added to platform 100 (e.g., via download orupdate or patch) and attachment manager 110 inspect attachment manifest150 for that component to see if any drivers, processes, or UICs need tobe loaded. When a software component is added to platform 100,attachment manager 110 can also choose to load the software componentautomatically without user input.

Several other simple examples are shown in FIGS. 4A, 4B, and 5 : (a) Theuser attaches a new gripper to robot 310, and the robot UI is updatedwith an OPEN and CLOSE buttons for that gripper automatically; (b) Theuser downloads a new perception software module, and attachment manager110 automatically loads the user interface for that module; (c) The userremoves an active peripheral electric gripper from robot 310, and thenattaches a passive, air-driven peripheral. Attachment manager 110responds by unloading all drivers and UICs for the electric gripper andthen load an air valve driver for the air gripper, as well as UICs forthe air control. In this manner, a UIC can be used for more than onecomponent; (d) The user inserts a spacer between end effector 315 ofrobot 310 and an attached gripper. The user drags this spacer to agraphical representation of end effector 315 using attachment MUI 115,and attachment manager 110 updates the mass and collision properties ofrobot 310 to compensate for the increased mass and length of the endeffector loadout; and (e) The user connects a foot pedal actuator viawire to robot 310. The user specifies this connection using attachmentMUI 115, and attachment manager 110 loads the driver for the actuator.Attachment manager 110 does not update the mass or collision propertiesof robot 310 as the pedal actuator is not physically connected to robot310.

FIG. 6 illustrates a computer system 600 that is consistent withembodiments of the present disclosure. In general, embodiments ofsystems and methods for connection-driven modification of robots'properties and generation of robotic user interfaces may be implementedin or performed by various computer systems, such as one or morepersonal computers, servers, workstations, embedded systems,multifunction devices, or a combination thereof. Certain embodiments ofthe systems or modules therein may be embedded as a computer program.The computer program may exist in a variety of forms both active andinactive. For example, the computer program can exist as softwareprogram(s) comprised of program instructions in source code, objectcode, executable code or other formats; firmware program(s); or hardwaredescription language (“HDL”) files. Any of the above can be embodied ona computer readable medium, which include storage devices and signals,in compressed or uncompressed form. However, for purposes ofexplanation, system 600 is shown as a general purpose computer that iswell known to those skilled in the art. Examples of the components andperipherals that may be included in system 600 will now be described.

As shown, system 600 may include at least one processor 602, a keyboard617, a pointing device 618 (e.g., a mouse, a 3-D pointing device, atouchpad, and the like), a display 616, main memory 610, an input/outputcontroller 615, and a storage device 614. Storage device 614 cancomprise, for example, RAM, ROM, flash memory, EEPROM, CD-ROM or otheroptical disk storage, magnetic disk storage or other magnetic storagedevices, or any other medium that can be used to carry or store desiredprogram code in the form of instructions or data structures and that canbe accessed by a computer. A copy of the computer program embodiment ofthe printer driver can be stored on, for example, storage device 614.System 600 may also be provided with additional input/output devices,such as a printer (not shown). The various components of system 600communicate through a system bus 612 or similar architecture. Inaddition, system 600 may include an operating system (“OS”) 620 thatresides in memory 610 during operation. One skilled in the art willrecognize that system 600 may include multiple processors 602. Forexample, system 600 may include multiple copies of the same processor.Alternatively, system 600 may include a heterogeneous mix of varioustypes of processors. For example, system 600 may use one processor as aprimary processor and other processors as co-processors. For anotherexample, system 600 may include one or more multi-core processors andone or more single core processors. Thus, system 600 may include anynumber of execution cores across a set of processors (e.g., processor602). As to keyboard 617, pointing device 618, and display 616, thesecomponents may be implemented using components that are well known tothose skilled in the art. One skilled in the art will also recognizethat other components and peripherals may be included in system 600.

Main memory 610 serves as a primary storage area of system 600 and holdsdata that is actively used by applications, such as the attachmentmanager in the robotic programming platform, running on processor 602.One skilled in the art will recognize that applications are softwareprograms that each contains a set of computer instructions forinstructing system 600 to perform a set of specific tasks duringruntime, and that the term “applications” may be used interchangeablywith application software, application programs, device drivers, and/orprograms in accordance with embodiments of the present teachings. Memory610 may be implemented as a random access memory or other forms ofmemory as described below, which are well known to those skilled in theart.

OS 620 is an integrated collection of routines and instructions that areresponsible for the direct control and management of hardware in system600 and system operations. Additionally, OS 620 provides a foundationupon which to run application software and device drivers. For example,OS 620 may perform services, such as resource allocation, scheduling,input/output control, and memory management. OS 620 may be predominantlysoftware, but may also contain partial or complete hardwareimplementations and firmware. Well known examples of operating systemsthat are consistent with the principles of the present teachings includeROBOT OPERATING SYSTEM, LINUX, UNIX, ORACLE SOLARIS, MICROSOFT WINDOWS,MAC OS, OPEN VMS, and IBM AIX.

The foregoing description is illustrative, and variations inconfiguration and implementation may occur to persons skilled in theart. For instance, the various illustrative logics, logical blocks,modules, and circuits described in connection with the embodimentsdisclosed herein may be implemented or performed with a general purposeprocessor (e.g., processor 602), an application specific integratedcircuit, a field programmable gate array or other programmable logicdevice, discrete gate or transistor logic, discrete hardware components,or any combination thereof designed to perform the functions describedherein. A general-purpose processor may be a microprocessor, but, in thealternative, the processor may be any conventional processor,controller, microcontroller, or state machine. A processor may also beimplemented as a combination of computing devices, e.g., amicroprocessor, a plurality of microprocessors, or any other suchconfiguration.

In one or more exemplary embodiments, the functions described may beimplemented in hardware, software, firmware, or any combination thereof.For a software implementation, the techniques described herein can beimplemented with modules (e.g., procedures, functions, subprograms,programs, routines, subroutines, modules, software packages, classes,and so on) that perform the functions described herein. A module can becoupled to another module or a hardware circuit by passing and/orreceiving information, data, arguments, parameters, or memory contents.Information, arguments, parameters, data, or the like can be passed,forwarded, or transmitted using any suitable means including memorysharing, message passing, token passing, network transmission, and thelike. The software codes can be stored in memory units and executed byprocessors. The memory unit can be implemented within the processor orexternal to the processor, in which case it can be communicativelycoupled to the processor via various means as is known in the art.

If implemented in software, the functions may be stored on ortransmitted over a computer-readable medium as one or more instructionsor code. Computer-readable media includes both tangible, non-transitorycomputer storage media and communication media including any medium thatfacilitates transfer of a computer program from one place to another. Astorage media may be any available tangible, non-transitory media thatcan be accessed by a computer. By way of example, and not limitation,such tangible, non-transitory computer-readable media can comprise RAM,ROM, flash memory, EEPROM, CD-ROM or other optical disk storage,magnetic disk storage or other magnetic storage devices, or any othermedium that can be used to carry or store desired program code in theform of instructions or data structures and that can be accessed by acomputer. Disk and disc, as used herein, includes CD, laser disc,optical disc, DVD, floppy disk and Blu-ray disc where disks usuallyreproduce data magnetically, while discs reproduce data optically withlasers. Also, any connection is properly termed a computer-readablemedium. For example, if the software is transmitted from a website,server, or other remote source using a coaxial cable, fiber optic cable,twisted pair, digital subscriber line (DSL), or wireless technologiessuch as infrared, radio, and microwave, then the coaxial cable, fiberoptic cable, twisted pair, DSL, or wireless technologies such asinfrared, radio, and microwave are included in the definition of medium.Combinations of the above should also be included within the scope ofcomputer-readable media.

Resources described as singular or integrated can in one embodiment beplural or distributed, and resources described as multiple ordistributed can in embodiments be combined. The scope of the presentteachings is accordingly intended to be limited only by the followingclaims. Although the invention has been described with respect tospecific embodiments, those skilled in the art will recognize thatnumerous modifications are possible. For instance, the proxy servers canhave additional functionalities not mentioned herein. In addition,embodiments of the present disclosure can be realized using anycombination of dedicated components and/or programmable processorsand/or other programmable devices. While the embodiments described abovecan make reference to specific hardware and software components, thoseskilled in the art will appreciate that different combinations ofhardware and/or software components can also be used and that particularoperations described as being implemented in hardware might also beimplemented in software or vice versa.

What is claimed is:
 1. A computer-implemented method for generation of arobotic graphical user interface, the method comprising: detecting arobotic peripheral attachment event for a robot and a roboticperipheral; obtaining from an attachment manifest a peripheral propertyset corresponding to the robotic peripheral, wherein the attachmentmanifest specifies respective peripheral property sets for a pluralityof robotic peripherals capable of being connected to the robot, andwherein the peripheral property set represents one or more properties ofthe robotic peripheral; and generating a robotic graphical userinterface during runtime, based on the peripheral property set, inresponse to the detecting the robotic peripheral attachment event;wherein the robotic peripheral is controllable in response to user inputreceived via the robotic graphical user interface.
 2. The method ofclaim 1, further comprising receiving, from one of the roboticperipheral or an attachment manager user interface, a peripheralidentifier of the robotic peripheral, wherein the obtaining theperipheral property set corresponding to the robotic peripheralcomprises obtaining the peripheral property set based on the peripheralidentifier.
 3. The method of claim 1, wherein the robotic peripheralattachment event comprises an establishment of a wired connectionbetween the robot and the robotic peripheral.
 4. The method of claim 1,wherein the robotic peripheral attachment event comprises auser-specified robotic peripheral attachment event.
 5. The method ofclaim 1, wherein the robotic peripheral attachment event comprises atleast one of: a robotic software module being enabled or disabled forthe robot, or a robotic software module enabled for the robot beingselected or deselected.
 6. The method of claim 1, further comprisingcontrolling the robotic peripheral in response to user input receivedvia the robotic graphical user interface.
 7. The method of claim 1,wherein the generating a robotic graphical user interface during runtimecomprises generating the robotic graphical interface in real time. 8.The method of claim 1, further comprising modifying, based on theperipheral property set, a robotic property set that represents one ormore properties of the robot to provide a modified robotic property set,wherein the one or more properties of the robot include at least one ofmass-related information, inertia-related information, dynamics-relatedinformation, collision-related information, control gains-relatedinformation, simulation-related information, logical state-relatedinformation, kinematics-related information, connection site-relatedinformation, or one or more images representative of the robot.
 9. Anon-transitory computer readable medium comprising instructions that,when executed by an electronic processor, cause the electronic processto generate a robotic graphical user interface by performing operationscomprising: detecting a robotic peripheral attachment event for a robotand a robotic peripheral; obtaining from an attachment manifest aperipheral property set corresponding to the robotic peripheral, whereinthe attachment manifest specifies respective peripheral property setsfor a plurality of robotic peripherals capable of being connected to therobot, and wherein the peripheral property set represents one or moreproperties of the robotic peripheral; and generating a robotic graphicaluser interface during runtime, based on the peripheral property set, inresponse to the detecting the robotic peripheral attachment event;wherein the robotic peripheral is controllable in response to user inputreceived via the robotic graphical user interface.
 10. Thenon-transitory computer readable medium of claim 9, wherein theoperations further comprise receiving, from one of the roboticperipheral or an attachment manager user interface, a peripheralidentifier of the robotic peripheral, wherein the obtaining theperipheral property set corresponding to the robotic peripheralcomprises obtaining the peripheral property set based on the peripheralidentifier.
 11. The non-transitory computer readable medium of claim 9,wherein the robotic peripheral attachment event comprises anestablishment of a wired connection between the robot and the roboticperipheral.
 12. The non-transitory computer readable medium of claim 9,wherein the robotic peripheral attachment event comprises auser-specified robotic peripheral attachment event.
 13. Thenon-transitory computer readable medium of claim 9, wherein the roboticperipheral attachment event comprises at least one of: a roboticsoftware module being enabled or disabled for the robot, or a roboticsoftware module enabled for the robot being selected or deselected. 14.The non-transitory computer readable medium of claim 9, wherein theoperations further comprise controlling the robotic peripheral inresponse to user input received via the robotic graphical userinterface.
 15. A computer system for generation of a robotic graphicaluser interface, the computer system comprising at least one electronicprocessor that executes instructions to perform operations comprising:detecting a robotic peripheral attachment event for a robot and arobotic peripheral; obtaining from an attachment manifest a peripheralproperty set corresponding to the robotic peripheral, wherein theattachment manifest specifies respective peripheral property sets for aplurality of robotic peripherals capable of being connected to therobot, and wherein the peripheral property set represents one or moreproperties of the robotic peripheral; and generating a robotic graphicaluser interface during runtime, based on the peripheral property set, inresponse to the detecting the robotic peripheral attachment event;wherein the robotic peripheral is controllable in response to user inputreceived via the robotic graphical user interface.
 16. The computersystem of claim 15, wherein the operations further comprise receiving,from one of the robotic peripheral or an attachment manager userinterface, a peripheral identifier of the robotic peripheral, whereinthe obtaining the peripheral property set corresponding to the roboticperipheral comprises obtaining the peripheral property set based on theperipheral identifier.
 17. The computer system of claim 15, wherein therobotic peripheral attachment event comprises an establishment of awired connection between the robot and the robotic peripheral.
 18. Thecomputer system of claim 15, wherein the robotic peripheral attachmentevent comprises a user-specified robotic peripheral attachment event.19. The computer system of claim 15, wherein the robotic peripheralattachment event comprises at least one of: a robotic software modulebeing enabled or disabled for the robot, or a robotic software moduleenabled for the robot being selected or deselected.
 20. The computersystem of claim 15, wherein the operations further comprise controllingthe robotic peripheral in response to user input received via therobotic graphical user interface.